feat(bundles): metapackage split (Phase A) — engine-only lfx, lfx-bundles long tail, 5 graduated partner packages by erichare · Pull Request #13563 · langflow-ai/langflow

erichare · 2026-06-09T17:41:45Z

Bundle Separation Phase A — metapackage split (1.11) · meta PR

This is the single PR for the entire Phase A bundle-separation epic. The original 10-PR stack has been fully collapsed into this branch as each ticket passed review — #13564, #13565, #13566, #13567, and #13568 merged here directly, and #13568 itself carried #13573, #13576, #13577, #13578, #13579, and #13580. Only this PR remains to merge into release-1.11.0.

Stacked on this PR: #13502 (Oracle integration) is based on this branch — it arrives as a sixth graduated lfx-oracle bundle following the same conventions (manifest-shipping, bounded ranges, .dev0 lfx floor, migration-table entries, pilot-upgrade test) and merges after this PR lands.

What it does: lfx becomes engine-only. The ~45-provider long tail moves into one manifest-less lfx-bundles metapackage discovered via a new lfx.bundles entry-point group, and five partner/flagship providers graduate to standalone manifest-shipping packages: lfx-openai, lfx-anthropic, lfx-amazon, lfx-datastax, lfx-cohere. Zero user-facing change: ext:<provider>:<Class>@official component ids are unchanged, in-tree shims keep from lfx.components.<provider> import X working, and existing flows load and run identically.

~328 files changed (+39k / −59k with rename detection; most of the delta is providers relocating from src/lfx/src/lfx/components/ into src/bundles/).

Part 1 — manifest-less `lfx.bundles` discovery + precedence tier (foundation)

New load_lfx_bundles_extensions (src/lfx/src/lfx/extension/loader/_bundles_root.py) iterates the lfx.bundles setuptools entry-point group, resolves each declared package via find_spec, folder-walks its immediate subdirectories, and registers each as a bundle at the @official slot with no extension.json (the langchain-community model). Reuses _load_bundle_directory, mirrors discover_inline_bundles.
Discovery precedence is now installed > seed > lfx_bundles > dev > inline. Manifest always wins: a manifest-shipping lfx-<provider> shadows the same-named metapackage provider with a typed bundle-shadowed warning, and the shadowed provider is never imported — this is what lets a provider graduate with no lockstep release.
New bundle-discovery-malformed error code, emitted on warnings (never flips ok / aborts startup), with per-mode codes for unresolvable declarations, plain-module targets, namespace-package portions, and invalid provider folder names. Manifest-less bundles get a typed reload refusal instead of a silent no-op.

Part 2 — `lfx-bundles` metapackage skeleton (#13564) + 45-provider move (#13568)

src/bundles/lfx-bundles/: single pyproject.toml declaring the lfx.bundles entry-point, a pre-release-safe lfx>=1.11.0.dev0,<2.0.0 floor, a generated all extra, and a README documenting the model + install stories. Hyphen dir name so release.yml's src/bundles/*/pyproject.toml glob builds it with zero workflow change.
45 long-tail providers (chroma, groq, ollama, mistral, qdrant, tavily, mem0, …) move from src/lfx/src/lfx/components/ into src/bundles/lfx-bundles/src/lfx_bundles/, each discovered manifest-less at @official. In-tree marker shims preserve the old import paths; the migration table and component_index.json are updated append-only/surgically with sha256 re-verified.
Removes the orphaned scripts/ci/update_bundle_versions.py (+ tests) — the nightly bundle-rename track it served was retired by the canonical pre-release cutover (src/bundles/NIGHTLY.md).

Part 3 — engine-only `lfx` + `lfx[bundles]` extra (#13565)

pip install lfx → engine only; pip install "lfx[bundles]" → engine + the lfx-bundles long tail. Intentionally no lfx[all]. lfx-bundles appears only under extra == "bundles", never in core Requires-Dist.

Part 4 — partner graduations (#13573)

lfx-openai, lfx-anthropic, lfx-amazon, lfx-datastax, lfx-cohere: each ships extension.json (lfx.compat: ["1"]), a langflow.extensions entry-point, an lfx pin, and is a workspace member pinned by langflow. Partner set is disjoint from lfx-bundles (verified).
datastax (the heavy one): lfx.base.datastax → lfx_datastax.base, 74 backend tests move with it, 10 repo consumers rewritten (incl. the vector_store_rag starter project).
Validator fix surfaced by this work: validate_extension now accepts classes whose base is a derived Component base (LCVectorStoreComponent etc.) that inherit class-level outputs.
Lock diff adds only the five lfx-* package names — no third-party package added or removed.

Part 5 — CI: cross-bundle matrix (#13566) + components freeze gate (#13567)

.github/workflows/cross-bundle-test.yml: per (bundle × Python 3.10/3.13) — installs in-repo lfx + the bundle, imports the entry-point package, asserts lfx.bundles discovery is error-free, runs lfx extension validate for manifest bundles, and runs the bundle's own tests/. Discovers bundles via the same glob release.yml uses, so each new extraction self-registers. Triggers: pull_request, workflow_dispatch, weekly schedule, workflow_call.
scripts/ci/check_components_frozen.py + baseline frozen_component_dirs.txt: no new top-level provider directory may be added to src/lfx/src/lfx/components/ — new providers go to lfx-bundles or a graduated package. Removals are allowed (shim cleanup never trips it). Wired into extension-migration-checks.yml so it actually runs on PRs (re-homed from lint-py.yml during review).

Part 6 — hardening + review follow-ups (#13576–#13580 and branch commits)

Bounded version ranges for the curated lfx-* packages (chore(bundles): bounded version ranges for curated lfx-* packages #13576); shim-contract lock tests + breakage audit (test(bundles): lock the in-tree shim contract + breakage audit #13577); bundle docs — install shapes, override rule, bundle_api_version (docs(bundles): install shapes, override rule, bundle_api_version #13578); stack-review fixes — symlink containment, idempotency, docs accuracy (fix(bundles): stack-review fixes — symlink containment, idempotency, docs accuracy #13579); tombstone for the broken legacy ZepChatMemory build method (fix(bundles): tombstone the broken legacy ZepChatMemory build method #13580).
Post-collapse review fixes on this branch: restore legacy-name resolution for ext components + starter-project sync; keep importable module paths in starter projects; restore loading.py / tableAutoCellRender to base-branch state (out-of-scope changes reverted); frontend test updates.

Test plan

Extension suite green (463 tests): discovery + @official registration, malformed-declaration / plain-module / namespace-portion / invalid-name warnings, manifest-shadows-manifestless, typed reload refusal
All five graduated packages register via the installed-manifest tier (20 components); manifest-less lfx_bundles registers its providers, disjoint from the partner set
60 pilot-upgrade migration tests + 74 moved datastax tests pass; all shims import; engine-safe import lfx.components
lfx extension validate passes for all manifest bundles; component_index.json surgically updated, sha256 verified
uv sync resolves; wheels build; lfx METADATA carries lfx-bundles only under extra == "bundles"
Cross-bundle matrix green (all bundle × py3.10/3.13 legs); freeze gate passes on baseline, fails actionably on a simulated new provider dir
ruff + ruff format clean; mypy clean on new modules

Collapsed stack (all merged into this branch)

PR	Scope
#13564	`lfx-bundles` metapackage skeleton
#13565	`lfx[bundles]` extra, engine-only `lfx`
#13566	cross-bundle CI matrix
#13567	`lfx/components` freeze gate
#13568	45-provider bulk move (carried the PRs below)
#13573	graduate openai / anthropic / amazon / datastax / cohere
#13576	bounded version ranges for curated `lfx-*`
#13577	shim contract tests + breakage audit
#13578	docs: install shapes, override rule, `bundle_api_version`
#13579	stack-review fixes (symlinks, idempotency, docs)
#13580	tombstone broken legacy `ZepChatMemory` build

Summary by CodeRabbit

New Features
- Added manifest-less bundle discovery for distribution metapackages through entry-point registration
- Immediate subdirectories are now automatically recognized as bundles without requiring configuration files
- Updated bundle discovery precedence order: installed > seed > lfx_bundles > dev > inline
- Added warning-only error handling for malformed bundle declarations that won't interrupt startup

@official

… tier Foundation for the bundle metapackage split (1.11). Adds a third @official-slot discovery source: a distribution declaring the [project.entry-points."lfx.bundles"] group ships a package whose immediate subdirectories are each a manifest-less bundle, folder-walked and registered at @official with no extension.json (the langchain-community model). - new loader/_bundles_root.py::load_lfx_bundles_extensions, mirroring discover_inline_bundles but sourcing roots from the lfx.bundles entry-point group via find_spec and reusing _load_bundle_directory - discovery precedence becomes installed > seed > lfx_bundles > dev > inline so a manifest-shipping lfx-<provider> always shadows the same-named provider in the metapackage (lets a provider graduate with no lockstep release), emitting the existing bundle-shadowed warning - new bundle-discovery-malformed warning code, emitted on warnings (never flips ok / aborts startup) for unresolvable declarations and invalid provider folder names - exported via loader/__init__ and the lfx.extension PEP-562 lazy surface 10 new tests in test_load_lfx_bundles.py; full extension unit suite (449) passes; ruff + format clean; mypy clean on the new module.

coderabbitai · 2026-06-09T17:41:52Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4c7272f4-0667-4e8d-85de-802952b14458

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Walkthrough

This PR adds manifest-less bundle discovery for the lfx.bundles entry-point group. The implementation resolves entry-point packages to filesystem directories, enumerates subdirectories as bundles registered at the @official slot, and integrates into the existing five-source discovery precedence with warning-only error handling for malformed declarations.

Changes

Manifest-less Bundle Discovery

Layer / File(s)	Summary
Error codes and public API exposure `src/lfx/src/lfx/extension/errors.py`, `src/lfx/src/lfx/extension/__init__.py`, `src/lfx/src/lfx/extension/loader/__init__.py`	New `bundle-discovery-malformed` warning-only error code with message template; `load_lfx_bundles_extensions` exported from `lfx.extension` and `lfx.extension.loader` via `__all__` and lazy-loading mechanisms.
Core manifest-less bundle loading `src/lfx/src/lfx/extension/loader/_bundles_root.py`	Resolves `lfx.bundles` entry-points to filesystem directories without importing, enumerates immediate subdirectories as bundle providers, validates names, handles duplicates via first-wins precedence, and emits warning-only `bundle-discovery-malformed` sentinels for resolution failures and invalid directory names.
Bundle discovery precedence and shadowing `src/lfx/src/lfx/interface/components.py`	Updates `_DISCOVERY_PRECEDENCE` to include `lfx_bundles` between `seed` and `dev`; extends `_resolve_bundle_shadowing` to a five-source deduplication tuple; calls `load_lfx_bundles_extensions()` in `import_extension_components` and populates the registry with deduplicated results.
Unit tests and error snapshots `src/lfx/tests/unit/extension/loader/test_load_lfx_bundles.py`, `src/lfx/tests/unit/extension/test_errors.py`	Tests bundle discovery, provider enumeration, entry-point resolution with fallback error handling, cross-source precedence/shadowing with installer-manifest and inline-source collisions, and error formatting snapshots for `bundle-discovery-malformed`.
API changelog documentation `BUNDLE_API.md`	Documents manifest-less `lfx.bundles` discovery, `load_lfx_bundles_extensions` export, five-source precedence, warning-only error code, and metapackage PEP 508 compatibility gating.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Suggested labels

lgtm

Suggested reviewers

dkaushik94
ogabrielluiz
Cristhianzl

🚥 Pre-merge checks | ✅ 8 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title mentions 'metapackage split (Phase A)' and 'engine-only lfx, lfx-bundles long tail, 5 graduated partner packages' but the actual changeset focuses narrowly on implementing manifest-less lfx.bundles discovery and precedence tier—not a full metapackage split or 5 graduated packages.	Revise the title to accurately reflect the core change: 'feat(extension): manifest-less lfx.bundles discovery and precedence tier' or similar, as this PR is Phase A foundation work that enables the broader metapackage split rather than implementing it.

✅ Passed checks (8 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 91.67% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Test Coverage For New Implementations	✅ Passed	PR includes 10 comprehensive unit tests for new load_lfx_bundles_extensions covering discovery, registration, error handling, shadowing, and precedence with 34+ assertions and no placeholder tests.
Test Quality And Coverage	✅ Passed	10 tests, 34 assertions. Comprehensive pytest patterns. Validates bundle registration, malformed handling, deduplication, precedence - not smoke tests.
Test File Naming And Structure	✅ Passed	test_load_lfx_bundles.py follows pytest backend convention: test_*.py name, 10 descriptive tests with docstrings, uses fixtures, helper functions, covers positive/negative/edge cases comprehensively.
Excessive Mock Usage Warning	✅ Passed	Test file uses zero mock objects from unittest.mock or pytest-mock. Tests use simple test doubles and real file construction with appropriate design patterns—no excessive mocking detected.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch bundles/foundation-discovery

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

…ndation-discovery

The BUNDLE_API.md changelog gate diffs each PR's branch against main, so the entry covering this PR's surface additions (lfx.bundles discovery, the lfx_bundles precedence tier, bundle-discovery-malformed) must live on this branch, not only further up the stack. Text is verbatim from the graduate-partners commit so the stacked merge dedupes trivially.

…ndation-discovery

github-actions · 2026-06-10T20:42:07Z

✅ Test Coverage Advisor

No source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉

Advisory check only — never blocks merge.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@BUNDLE_API.md`:
- Around line 405-421: Update the BUNDLE_API.md wording for manifest-less
lfx.bundles to avoid implying the CLI can validate those directories: change
“exempt from `lfx extension validate`” to something like “not validated / not
intended as input to `lfx extension validate`” and add a one-line explanation
that validate_extension() (the CLI check) requires an extension.json or
[tool.langflow.extension] in pyproject.toml so pointing the validator at a
manifest-less provider will report “manifest-not-found”; reference the loader
module src/lfx/src/lfx/extension/loader/_bundles_root.py and the
validate_extension() CLI behavior in the note.

In `@src/lfx/src/lfx/extension/loader/_bundles_root.py`:
- Around line 246-260: _spec_package_dir currently treats plain module specs as
package roots by returning Path(spec.origin).parent when
submodule_search_locations is absent; change it to reject non-package
entry-point targets by removing the fallback that returns the origin's parent.
Update _spec_package_dir so it only returns a Path when
spec.submodule_search_locations is present (i.e., a package or namespace
package) and otherwise returns None; keep the initial spec is None guard and do
not treat spec.origin as a package root.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: adefeccc-6f5e-4660-b680-b1263385f836

📥 Commits

Reviewing files that changed from the base of the PR and between 29140c0 and 5bec743.

📒 Files selected for processing (8)

BUNDLE_API.md
src/lfx/src/lfx/extension/__init__.py
src/lfx/src/lfx/extension/errors.py
src/lfx/src/lfx/extension/loader/__init__.py
src/lfx/src/lfx/extension/loader/_bundles_root.py
src/lfx/src/lfx/interface/components.py
src/lfx/tests/unit/extension/loader/test_load_lfx_bundles.py
src/lfx/tests/unit/extension/test_errors.py

…date wording CodeRabbit review fixes on #13563: - _spec_package_dir no longer falls back to a plain module's parent directory -- a single-file entry-point target now emits the bundle-discovery-malformed warning instead of folder-walking unrelated sibling directories as bundles (+ regression test) - BUNDLE_API.md / loader docstring no longer say manifest-less providers are 'exempt from lfx extension validate'; they are not valid validator input (validate requires a manifest and reports manifest-not-found)

Commit 9df9475 ('Frontend test updates') accidentally reverted both files to their pre-#9902 state, removing the no_env_fallback contract in load_from_env_vars / update_table_params_with_load_from_db_fields, the defensive params.pop('code') in build_custom_component, and the localValue/updateGlobalVariableCell handling in tableAutoCellRender. The base branch's regression tests (test_loading_no_env_fallback.py, test_loading_custom_component_code_param.py, tableAutoCellRender index.test.tsx) were kept, so the reversion failed them.

- Move the freeze-components job from lint-py.yml (workflow_call/ workflow_dispatch only -- never invoked, so the gate could not block a PR) into extension-migration-checks.yml, which already triggers on pull_request for src/lfx/src/lfx/components/**. Add the gate's script and baseline to the path filter. - Add timeout-minutes to the cross-bundle-test jobs so a hung install or test leg cannot occupy a runner for the 360-minute default. - _discover_shimmed_component_dirs: read only the first line of each __init__.py instead of the whole file -- the shim marker contract is line 1 (enforced by test_shim_source_contract), and this prevents a non-shim file with the marker after leading blank lines from being misclassified and silently skipped from the in-tree walk.

@patch

… starter projects Saved flows reference moved providers by their legacy palette identity -- either the bare class name (TavilySearchComponent) or the component's name attribute (AstraDB, Chroma, needle, OllamaModel). After the provider move neither resolved to a current template on the backend: - lfx.utils.component_aliases now derives the bare class name (and its Component-stripped form) from ext:<bundle>:<Class>@<slot> keys, mirroring the frontend's getTemplateAliases. Ext templates carry name=None / _type='Component', so the key is the only source. - _decorate_template_with_extension now stamps template['name'] with the component's legacy identity (name attr, falling back to class name). In-tree palette entries expose it as the dict key; ext entries are keyed by namespaced id, so without this both alias maps lose the only bridge from node types like 'AstraDB' to the current template. Without these, the starter-project updater could not match moved-provider nodes, so their embedded code stayed one whitespace change behind the bundle sources -- every such flow showed the update-all banner, and because tool-mode nodes' saved outputs ([component_as_tool]) differ from the template's natural outputs the update was flagged breaking, opening the confirmation modal instead of toasting 'successfully updated': deterministic timeouts in the starter-project Playwright specs (shards 29/30/36/37). Regenerated the 13 affected starter projects with scripts/ci/update_starter_projects.py (idempotent; code_hash hex strings in the regenerated JSONs are detect-secrets false positives). Also converts test_groq_integration.py to the captured-module patch.object pattern (same module-identity trap as test_mongodb_atlas: dotted-name @patch can land on a different materialization of the moved module than the one that defined the class, leaving the real get_groq_models in the instance's globals -- order-dependent failure in unit-test Group 5).

The starter-project updater syncs node metadata from the live templates (NODE_FORMAT_ATTRIBUTES includes 'metadata'), and ext components report their runtime sys.modules namespace there (_lfx_ext.official.<p>.<m>) -- a path that only exists inside a running extension loader. Persisting it broke test_template_field_order_matches_component in both template jobs ('No module named _lfx_ext'): the test imports metadata.module to instantiate the component, and the migration table likewise keys on the legacy lfx.components.<provider> form that the bundle shims keep importable. _merge_node_metadata now preserves the node's stored module when the live template's is a runtime _lfx_ext.* path, while still syncing the rest of the metadata (e.g. dependencies). Starter projects regenerated from the pre-pollution state with the fixed updater: the only diff vs the previous regen is the 20 module lines reverting to the legacy importable form. Regression tests added for both merge directions.

@official

* Add Oracle Integration * [autofix.ci] apply automated fixes * Fix styleUtils problem * fix(oracledb): address review feedback * fix(oracledb): apply formatting * Fix component index * refactor(oracledb): convert Oracle integration to lfx-oracle extension bundle Port the in-tree lfx.components.oracledb provider into a standalone Extension Bundle distribution (src/bundles/oracle, dist lfx-oracle, package lfx_oracle, bundle name 'oracle') per src/bundles/PORTING.md: - Move the four components + connection helper into the bundle; deps (oracledb, langchain-oracledb, langchain-community) move from the langflow-base[oracledb] extra into the bundle's pyproject - Add migration-table entries mapping legacy bare class names and lfx.components.oracledb.* import paths to ext:oracle:<Class>@official - Add test_pilot_oracle_upgrade.py integration coverage (mirrors ibm) - Move backend unit tests into src/bundles/oracle/tests with bundle import paths; drop the empty ComponentTestBaseWithoutClient version-fixture scaffolding (component is new in 1.11.0) - Declare explicit outputs on OracleVectorStoreComponent (mirrors LCVectorStoreComponent) so lfx extension validate passes - Regenerate component index (oracledb category removed) and uv.lock; wire workspace member/dep/source in root pyproject - Frontend sidebar entry renamed oracledb -> oracle; docs page renamed to bundles-oracle following the extracted-bundle convention --------- Co-authored-by: autofix-ci[bot] <114827586+autofix-ci[bot]@users.noreply.github.com> Co-authored-by: Eric Hare <ericrhare@gmail.com>

github-actions · 2026-06-12T15:17:19Z

Build successful! ✅
Deploying docs draft.
Deploy successful! View draft

github-actions Bot added the enhancement New feature or request label Jun 9, 2026

erichare changed the base branch from feat/bundle-metapackage-split to release-1.11.0 June 9, 2026 20:45

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 9, 2026

Merge remote-tracking branch 'origin/release-1.11.0' into bundles/fou…

dbedb8f

…ndation-discovery

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 9, 2026

erichare added 2 commits June 10, 2026 11:07

Merge remote-tracking branch 'origin/release-1.11.0' into bundles/fou…

5bec743

…ndation-discovery

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

erichare requested a review from dkaushik94 June 10, 2026 20:39

erichare marked this pull request as ready for review June 10, 2026 20:40

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

erichare requested a review from ogabrielluiz June 10, 2026 20:43

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

coderabbitai Bot reviewed Jun 10, 2026

View reviewed changes

Comment thread BUNDLE_API.md Outdated

Comment thread src/lfx/src/lfx/extension/loader/_bundles_root.py Outdated

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 10, 2026

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jun 11, 2026